ON  MAKING  THE  SHORTLIST 
FOR  THE  SELECTION  OF  CANDIDATES 

BY 


INGRAM  OLKIN  and  MICHAEL  A.  STEPHENS 


TECHNICAL  REPORT  NO.  430 
JUNE  26,  1990 


Prepared  Under  Contract 
NOOO 14-89- J- 1627  (NR-042-267) 

For  the  Office  of  Naval  Research 


Herbert  Solomon,  Project  Director 


Accesion  For 

1 _ 

1 

i  NTIS 

CRA&I 

k/ 

1  DTIC 

TAB 

□ 

Unannounced 
Justification  _ 

□ 

_ 

Distribution  / 


Availability  Codes 


Avail  and/or 

Dist 

special 

Reproduction  in  Whole  or  in  Part  is  Permitted 
for  any  purpose  of  the  United  States  Government 


Approved  for  public  release;  distribution  unlimited. 


DEPARTMENT  OF  STATISTICS 
STANFORD  UNIVERSITY 
STANFORD,  CALIFORNIA 


ON  MAKING  THE  SHORTLIST 
FOR  THE  SELECTION  OF  CANDIDATES 


by- 

Ingram  Olkin  and  Michael  A.  Stephens 

1.  INTRODUCTION 

Bias  in  the  selection  of  candidates  for  employment  or  for  the  admission  to  college  has 
been  an  issue  of  considerable  public  interest.  Although  there  may  be  generail  consensus 
on  the  existence  of  bias,  there  is  little  consensus  on  a  definition  of  bias,  and  indeed  there 
are  a  variety  of  definitions.  Cole  (1973)  provides  an  extensive  exposition  of  a  number  of 
definitions  that  are  known  by  the  names  quota  model,  regression  model,  equal  risk  model, 
constant  ratio  model,  conditional  probability  model.  We  describe  but  one  of  these  models 
in  order  to  establish  how  our  analysis  differs  from  these. 

In  the  constant  ratio  model  Thorndike  (1971)  proposed  a  definition  of  fairness.  Sup¬ 
pose  there  are  two  groups  A  and  B,  a  predictor  variable  .Y,  and  a  criterion  variable  Y. 
The  model  requires  that  the  success  ratio  be  equal  to  the  selection  ratio.  This  model  is 
implemented  in  the  following  way.  Let  Xa  and  Xb  denote  selection  cutoff  points  to  be 
determined  for  the  two  groups,  and  let  c  denote  the  given  passing  cutoff  point  on  the  cri¬ 
terion.  The  first  requirement  is  that  the  equality 

Pa{Y  >c}  _  Pa{X  >  A-.4} 

^  ’  Pb{Y>c}  Pg{X>XB} 

hold. 

If  we  are  to  select  a  total  of  X  applicants,  where  jXa  and  Xg  are  the  number  of 
applicants  in  groups  A  and  B,  respectively,  then  we  simultaneously  solve  (1)  and 

(2)  XaPa{X  >  A'.4}  +  XgPgiX  >  Xg)  =  A. 

for  -Y.^  and  Xg  for  a  given  passing  cutoff  point  c.  Often  these  computations  are  made 
with  the  assumption  that  in  each  group  (A'.T)  has  a  bivariate  normal  distribution  with 
known  mean  vectors  ha  and  and  with  known  covariance  matrices  11.4  and  Es- 
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A  fundamental  assumption  in  the  models  reviewed  by  Cole  ( 1973)  is  that  the  measure¬ 
ments  of  the  applicants  are  independently  and  identically  distributed  random  variables. 
These  models  do  not  mahe  use  of  the  ordered  scores  of  individuals,  which  is  the  starting 
point  in  our  analysis.  The  ordered  measurements  arise  in  many  applications.  For  example, 
in  the  physical  or  biological  sciences,  we  may  have  measurements  on  two  species,  eind  only 
the  largest  is  chosen  for  a  particular  purpose.  Similarly,  in  the  selection  of  candidates  for 
employment,  the  order  statistics  are  used  to  make  a  selection. 

More  specifically,  suppose  that  n  values  of  x  and  m  values  of  y  are  given;  the  values 
are  random  samples  from  continuous  distributions  Fi(i)  and  Fjly)  respectively.  The  x,y 
\’alues  could,  for  example,  be  thought  of  as  test  scores  for  two  groups  of  students.  The 
m  -f  n  scores  are  pooled  and  jointly  ranked  zind  the  top  k  students  are  picked  for  a  shorthst 
for  scholarships.  Our  concern  is  with  the  determination  of  the  probability,  P{r,k;n,m], 
that  exactly  r  students  from  the  first  group,  with  r-scores,  appear  in  the  shortlist. 

When  Fi(-)  ~  F2(-),  then  r  has  a  hypergeometric  distribution  and 


P{r,  k;  n,m} 


n  +  m\ 

k  ) 


For  the  interesting  special  case  of  choosing  a  single  candidate,  that  is,  r  =  A;  =  1, 


P{l.l;n,m}  =  P{x(„)  >  y(,„)}  =  n/(n  -l-m). 


where  X(i)  <  •  ■  •  <  X(n)  and  y(i)  <  ■  •  •  <  y(m)  are  the  order  statistics  of  samples  of  sizes  n 
and  m  from  identical  populations. 

When  n  is  small  relative  to  m,  this  probability  is  small.  Consequently,  even  when  all 
the  candidates  come  from  identical  populations,  the  probability  that  the  best  candidate 
from  a  small  unit  is  chosen  is  small.  This  suggests  that  an  individual  who  attends  a  small 
prestigious  school  may  have  considerably  less  chance  of  competing  with  an  individual  from 
a  larger  prestigious  school.  (It  remains  to  be  determined  whether  women  or  minorities 
generally  attend  smaller  schools  than  men.) 

This  result  also  may  explain  the  choice  of  one  of  two  candidates,  each  rated  as  “the 
best  candidate  we  have  had  in  the  last  ten  years.’’  where  the  only  difference  is  in  the  size 
of  the  pool  from  which  each  individual  is  drawn. 
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We  study  this  model  for  general  Filx)  and  F2(y).  Numerical  computations  can  be 
made  with  the  aid  of  Theorem  1.  In  particular,  we  consider  the  case  where  x  and  y 
have  normal  distributions  with  a  common  variance  1,  but  different  means,  and  provide 
some  tables  of  probabilities  for  selected  sample  sizes,  length  of  shortlist,  and  values  of  the 
mean  difference  ^  in  the  two  normal  populations.  What  this  points  out  is  the  relationship 
between  differences  in  means  versus  differences  in  sample  sizes. 


2.  PROBABILITIES 


2.1  Calculation  of  probabilities  for  identical  parent  populations. 

Formally,  the  problem  can  be  stated  as  follows.  Suppose  —  ^(2)  ^  £  ^(n) 

the  order  statistics  of  a  sample  of  size  n  from  Fi{x)  and  y(j)  <  y(2)  <  ■  •  •  <  y(m)  are  the 

order  statistics  of  an  independently  chosen  sample  of  size  m  from  F2{y).  The  two  samples 
are  pooled  and  ranked,  and  we  want  to  determine  P{r,  k‘,n,m),  abbreviated  P*,  which  is 
the  probability  that  r  of  the  x-values  axe  found  in  the  top  k  rankings. 

The  process  of  recruiting  usually  involves  the  reduction  of  a  list  of  candidates  to  a 
short  list  of  k  candidates.  Let  E{n,m;k)  denote  the  event  that  the  best  i-candidate  with 
score  i(n)  is  included  in  the  top  group  of  k  candidates;  thus  at  least  one  x-value  is  in  the 
list.  Then  P{E(n,  m\  k)}  =  1  —  P(0,  k\  n.  m)  =  1  —  {m{m  —  l)...(m  —  1’  +  -t- 

n)(m  +  n  —  1) . . .  {m  +  n  —  k  +  1)} ,  which  can  also  be  written 


P{Ein,nr.k)]  = 


n  +  m  —  1\  /n  +  m  —  2 

+ 


n  —  1 


n  —  1 


+ - h 


n  +  m  —  k 
n  —  1 


n  +  m 
n 


(2) 


This  formula  arises  naturally  from  considering  the  probability  that  X(n)  is  the  top  A  the 
list,  then  that  X(„)  is  second,  and  so  on  to  the  probability  that  j(„)  is  the  k-th  largest. 

A  numerical  example  may  be  informative.  Suppose  that  there  are  n  =  3  women 
candidates  and  m  =  25  men  candidates,  for  which  a  short  list  of  k  candidates  is  to  be 

chosen.  The  probability.  P,  that  at  lezist  one  woman  will  be  a  finalist  is.  for  k  =  1(1)10. 

/t:  1  2  3  4  5  6  7  S  C  10 

P:  .107  .206  .298  .382  .459  .530  .594  .652  .762  .809 


This  shows  that  with  these  values  of  m  and  n  we  need  a  sliort  list  of  size  at  least  A'  =  6  in 
order  to  guarantee  that  the  probability  of  at  least  one  woman  being  included  in  the  short 
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list  is  at  least  0.5.  Such  a  computation  provides  a  guide  whether  the  candidate  from  the 
smaller  group  has  an  opportunity  to  be  chosen  on  the  short  list. 

2.2  Calculation  of  P{r,k\n.m}  in  the  general  case. 

When  Fi(-)  ^  FzC-)  the  probability  P*  cannot  be  found  from  a  simple  combinatoric 
argument.  We  calculate  P*  by  considering  the  relationships  between  the  ranked  x  and  y 
values.  The  result  is  stated  in  Theorem  1  below. 

It  is  intuitively  more  appealing  to  relabel  the  x  and  y  sets  as  follows.  Let  A’l  = 
^(n)i  ^2  ^(n  —  !)!■••'  An  1 )  ^tld  let  Ij  —  J/(m)5^2  —  I/fm  —  —  y{l)i  that 

Ai  is  the  largest  r.  X2  is  the  second  largest,  and  so  on.  and  similarly  for  the  Y  values. 
Suppose,  to  fix  idea^,  that  n  =  8,  m  =  10,  and  the  shortlist  has  length  k  =  5.  Then,  in 
ascending  order  the  v'alues  might  finish  as 

...y^A'sVsA'aA'iVjli 

giving  r  =  2  values  A'  in  the  top  k  =  5  values.  For  r  to  be  equal  to  2,  it  must  be  true  that 
X2  >  ^41  for  if  not  there  would  be  at  least  4  values  in  the  top  5;  also  it  must  be  true 
that  5^3  >  A's,  or  there  would  be  3  A-values  in  the  top  5. 

Applying  the  above  argument  in  general  leads  to  Theorem  1. 

Theorem  1.  If  i(i)  <  •  •  •  <  r(„),  j/d)  <  •  •  •  <  then 

P  {r,  k .  n,  m}  =  P  {r(n4.i_r)  >  y(m+r— Jt)  }  ~  ^  {^(n  —  r)  >  y(m  +  r— t-t-l)}  •  (3) 

Proof.  By  an  analysis  as  given  earlier,  to  find  exactly  r  A’-values  in  the  top  k  values 
we  must  have  the  events  S  :  Xr  >  Yk+\-r  and  T  :  >  AA+i  jointly  true.  Thus 

P*  =  P{S  n  T'\  —  P{<f }  —  P{£  n  =  P{5}  —  P{!F],  where  T  denotes  the  complement 
of  T\  the  last  equality  follows  because  £  contains  P .  Thus 

P*  =  P  {Xr  >  -  P  {AV+I  >  y,_,} . 

which  completes  the  proof.  || 


4 


2.3  Calculations. 

A  cedculation  such  as  P  >  y(m+r-k)  }  is  difficult  to  make  accurately,  since  it 

involves  densities  of  two  families  of  order  statistics.  However,  if  the  moments  of  the  order 
statistics  axe  known,  an  excellent  approximation  can  be  found  as  follows. 

Suppose  Kj{x)  denotes  the  j-th  cumulant  of  a  \-ariable  x.  Let  w  =  X(,)  —  ?/(<);  then, 
because  X(,)  and  y(t)  are  from  independent  samples,  we  have 

K,{w)  =  K,  (i,.,)  +  (-lyA-,  (!/,„).  (5) 

If  the  moments  (or  cumulants)  of  X(j),  y^)  are  known  or  can  be  calculated,  the  first 
four  cumulants  of  w  can  be  found,  and  a  Pearson  curve  fitted  to  the  distribution  of  w\ 
then  P{w  >  0}  cein  be  easily  calculated,  as  required  for  Theorem  1. 

2.4  Normal  parent  populations. 

We  illustrate  these  computations  for  the  case  where  F\{x)  is  the  1)  distribution, 
and  where  F2{y)  is  the  .^(0, 1)  distribution;  Thus  the  distributions  of  x  and  y  are  normal, 
differing  only  in  means.  Let  x  =  z  +  ^  so  that  z  =  A''(0,1),  and  let  X(,)  =  Z(,)  +  /x. 
Then  P  {x(4)  >  y(()}  =  P  —  Z(,)  <  ^}.  The  moments  of  y^)  and  Z(3)  are  extensively 
tabulated  (Haxter,  1961)  or  can  easily  be  calculated  numerically.  These  have  been  used  to 
determine  P*  as  described  above,  for  vEirious  values  of  m.  n,  k,  r,  and  y..  Table  1  records 
k:  n.  m;  y.)  =  P{r  \'alues  of  x  in  shortlist  of  length  k},  when  there  are  n  values  of  x 
and  m  values  of  y;  x  has  distribution  1)  and  y  has  distribution  A*(0. 1).  The  values 
of  n  eire  5,  8,  12.  15.  20,  30:  m  >  n,  and  Py{r,  k;n.m:  fj.)  is  abbreviated  P^-.  Since  it  is 
easily  shown  that  PjJ)-  =  P^^’{k  —  r,  — /x).  Table  1  can  easily  be  used  when  n  >  m. 

For  example,  suppose  that  we  have  n  =  20  women  and  m  =  10  men.  with  a  shortlist  of 
length  k  =  4;  the  probability  of  exactly  3  women,  when  the  women's  mean  score  exceeds 
the  men’s  by  0.2  times  the  common  standard  deviation,  is 

Pv'(3,4:20,10;0.2)  =  P^d  1.4;  10,20; -0.2)  =  0.446. 

There  will  often  be  special  interest  in  the  case  when  n  is  small.  Table  2  gives  \’alues 
of  P^r[r.k\n,m\  ii)  for  all  combinations  n.m  where  n  =  1.  2.  3.  and  4  and  m  =  10,  15,  20. 
30. 


o 


2.5  A  check  on  numerical  accuracy. 

When  n  =  0,  the  distributions  of  x  and  y  are  identical  and  P^-  can  be  found  using 
the  hypergeometric  distribution  in  (1).  This  provides  a  check  on  the  accuracy  of  Tables  1 
and  2.  For  example,  for  n  =  m  =  5,  A:  =  2,  =  2/9,  5/9,  2/9  for  r  =  0.  1,  2  respectively. 

This  is  exactly  as  in  Table  1.  Table  3  gives  a  selection  of  values  of  calculated,  for 
r  =  0,  by  the  hypergeometric  formula,  and  also  those  given  by  Table  1.  It  can  be  seen 
that  the  Pearson  curve  fitting  gives  excellent  results.  Although  this  is  a  check  only  in  the 
middle  section  of  the  table,  past  experience  suggests  that  the  Pearson  curves  will  give  very 
close  approximations  to  true  distributions  when,  as  here,  these  do  not  have  steep  densities 
(Solomon  and  Stephens,  1978).  A  further  check  will  be  given  below. 

2.6  Use  of  Tables  1  and  2  with  other  normal  populations  with  equal  variances. 

Suppose  X  now  has  a  normal  distribution,  iV(^i,(T^),  and  y  has  a  normal  distribution 
.V(/i2,<T^),  and  we  again  want  Pfthere  are  r  \'alues  of  x  in  the  shortlist  of  length  k},  now 
called  P^*.  Suppose  U(,)  =  (r(,)  -  fii)/cr  and  y(,)  =  (t/(,)  -  fX2)/(r;  then  P{x(^)  >  = 

<  m},  where  n  =  (//i  -^2)/<7.  Since  U(,)  are  order 
statistics  from  a  standard  normal  distribution,  probabilities  of  the  last  type  can  be  used 
in  Theorem  1  to  give  Py  =  Py{r,k:n,m-,  fi).  Thus  Tables  1  and  2  can  be  used  to  find 
values  Py  for  normal  parent  populations  with  different  means  but  the  same  variance. 

2.7  Different  normal  populations,  with  n  =  1,  A:  =  1. 

This  corresponds  to  the  situation  in  which  the  single  A’-value  is  the  largest  of  all 
the  values.  For  this  situation  existing  tables  for  multivariate  analysis  can  be  used,  and 
probabilities  can  be  found,  for  the  more  general  problem  where  the  A’-value  comes  from  a 
completely  different  normal  distribution  from  that  of  the  }'-values.  Suppose  A'"  is  -I- 
A.crJ)  and  suppose  the  I'-values  come  from  Then  the  probability  P*  that  the 

A'  value  is  the  largest  of  all  the  values  can  be  expressed  as  P*  =  P{A'  —  I'l  >  0.  A  —  I2  > 
0. . . . ,  A'’  —  y'n  >0}.  The  typical  variable  IF,  =  A'  —  F',  is  A'^(A,(7‘),  where  cr^  =  cTj  -f  ctj, 
and  the  derived  variable  Z\  =  (IF,  —  A)/<7  is  A^O,  !)•  The  Z,  are  equi-correlated  with 
correlation  coefficient  p  =  ajfcr^.  Then 

P*  =  P(Zi  >  c,  Z2  >  c . Z„  >  c). 
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where  c  =  —  A/cr. 

Gupta  (1963,  Table  II)  has  given  a  table  from  which  P*  may  be  found.  Gupta’s  table 

gives  v'alues  of  G{H)  =  P(Zi  <  H,  Z2  <  H . Zn  <  H)  for  various  p  and  H.  and  clearly 

P-  =  G(-c). 

Example  1.  We  first  use  Gupta’s  table  to  make  a  check  on  our  Table  2.  Suppose  n  =  1 
and  m  =  10,  the  shortlist  has  length  fc  =  1.  the  excess  in  mean  for  A'  is  p  (=  A)  =  0.3, 
and  we  want  P*  =  P{r  =  1},  given  in  Table  2  as  0.142.  In  our  table,  aj  =  so  p  =  0.5; 
also  c  =  —A/a  =  — 0.3/\/2  =  —0.212.  We  enter  Gupta’s  Table  II  at  /f  =  — c  =  0.212,  and 
y  (=  our  m)  =  10.  Interpolation  in  the  table  gives  G(0.212)  =  0.142.  as  in  our  Table  2.  If 
p  =  1.  then  c  =  —0.707;  Table  II  gives  G(0.707)  =  0.323,  as  in  our  Table  2. 

Example  2.  Suppose  again  n  =  1,  m  =  10,  ^’  =  1,  but  now  a^  =  Then  a^  =  2.5cr|, 

and  p  =  0.6.  Suppose  A  =  0.3<72,  so  that  c  =  — 0.3<72/ {(2.5)^/^<T2  }  =  —0.190.  Entering 
Table  II  with  p  =  0.6,  N  =  10,  H  =  0.190  gives  P*  =  0.468.  The  checks  given  here, 
together  with  the  remarks  in  Section  2.5.  suggest  that  Tables  1  and  2  will  be  accurate 
enough  for  most  practical  purposes. 

2.8  Mean  exceedance  needed  to  obtain  at  least  one  A'. 

Table  2  can  be  extended,  and  interpolation  used,  to  answer  the  question:  how  large 
must  p  be  in  order  to  obtain  at  least  one  A’  on  the  shortlist,  with  a  certain  probability, 
say  P{TV'}?  This  would  often  be  of  interest  if  there  are  few  A’  values  compared  with  the 
number  of  Y  values.  Table  3  gives  values  of  p  for  n  =  1.  2,  3.  4  and  m  —  10.  15.  20,  30. 
to  ensure  at  least  one  X  in  the  list  of  length  k\  with  probability  P{ir}  =  O.SO,  or  0.90,  or 
0.95.  or  0.99.  Recall  that  p  represents  the  exceedance  in  (mean/standard  deviation)  of  A’ 
over  when  both  have  normal  populations  and  the  same  standard  deviation. 

Example  3.  Suppose  there  is  a  single  (n  =  1)  minority  candidate,  say.  and  m  =  10  non¬ 
minority  candidates.  Using  Table  3.  the  mean  of  the  minority  candidate  would  have  to  be 
2.05  standard  deviations  above  the  mean  of  the  non-minority  candidate  to  ensure,  with 
probability  0.90.  that  the  minority  candidate  would  be  on  a  short  list  of  size  3.  The  mean 
is  lowered  from  2.05  standard  deviations  to  1.76  standard  deviations  if  the  short  list  is  of 
size  4  instead  of  size  3:  that  is.  the  discrepancy  between  the  two  groups  diminishes  as  the 
size  of  the  short  list  increases. 


3.  FURTHER  DISCUSSION 

A  typicad  situation  where  calculations  similar  to  the  above  can  be  used  is  as  follows. 
Suppose  axi  advertisement  for  a  vacancy  yields  18  candidates  consisting  of  3  women  and 
15  men.  As  a  first  step  the  search  committee  will  prepare  a  shortlist  of  3  individuals  who 
are  to  be  interviewed.  If  all  individuals  are  from  the  same  parent  population  of  scores, 
then  the  probability  P{IF}  that  at  least  one  woman  will  be  a  finalist  is  0.442  from  (1)  or 
(2),  or  from  Table  2.  When  the  populations  have  different  means  it  is  interesting  to  know 
how  much  greater  must  be  the  mean  of  the  women  to  ensure,  say  a  probability  P{W} 
of  0.90.  From  Table  4,  with  n  =  3,  m  =  15,  ^  =  3,  P  =  0.90,  we  see  that  the  mean  of 
the  women's  distribution  must  exceed  the  men's  mean  by  1.15cr,  where  a  is  the  common 
standard  deviation,  to  achieve  a  P{W}  =  0.90. 
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Table  2.  Selected  values  of  the  probability  that  r  out  of  u  candidates  from  a  1)  i)()j)ulation, 
and  k  -  r  out  of  in  candidates  from  a  A/’(0, 1)  poi)iilatic)n,  are  chosen  on  a  short  list 
of  length  k.  (Case  n  small.) 
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APPENDIX 


In  this  Appendix  we  provide  more  extensive  tables  than  the  condensed  versions  in¬ 
cluded  in  the  body  of  the  paper. 

Table  1.  Selected  \’alues  of  the  probability  that  r  out  of  n  candidates  from  a  1) 
population,  and  k  —  r  out  of  m  candidates  from  a  .V’(0, 1)  population,  are  chosen  on  a  short 
list  of  length  k. 

Table  2.  Selected  values  of  the  probability  that  r  out  of  n  candidates  from  a  1) 
population,  and  k  —  r  out  of  m  candidates  from  a  A^(0, 1)  population,  are  chosen  on  a  short 
list  of  length  k.  (Case  n  small.) 

Table  3.  A’alue  of  /i,  the  excess  mean  in  a  A^(/i,  1)  population  over  a  A*(0, 1)  popu¬ 
lation  to  ensure  with  probability  P  that  at  least  1  out  of  n  candidates  from  the  A'*(^,  1) 
population  and  k  —  1  out  of  m  candidates  from  the  .V*(0, 1)  population  is  chosen  on  a  short 
list  of  length  k. 
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